IEEE Visualization (TVCG) 2025 AR/VR Gaze Tracking Temporal Modeling Seq2Seq Human-Computer Interaction

TimeGazer: Temporal Modeling of Predictive Gaze Stabilization for AR Interaction

Yaozheng Xia1   Zaiping Zhu2   Bo Pang3   Sheng Li3   Shaorong Wang1*

1 Beijing Forestry University, Beijing, China
2 Bournemouth University, National Centre for Computer Animation, Poole, United Kingdom
3 Peking University, Beijing, China

PDF arXiv

TL;DRThe first predictive gaze stabilization framework for AR that reformulates stabilization as a sequence-to-sequence temporal regression problem, leveraging historical gaze dynamics to refine fixation points without scene semantics or multimodal cues.

Abstract

TimeGazer reformulates gaze stabilization as a sequence-to-sequence temporal regression problem. It predicts idealized fixation trajectories for the target-fixation phase from historical gaze dynamics in the search phase. The model employs a TimesNet backbone with a triple embedding module (token, positional, timestamp) and a fused projection strategy (multi-head attention + linear), trained with a combined loss of MSE, center distance, and dispersion consistency. A synthetic data generation and blending strategy enriches the training space for improved generalization.

Method

Raw gaze data during active fixation tasks exhibits irregular dispersion and systematic deviations from target locations, caused by oculomotor physiology, insufficient AR headset tracking accuracy, and environmental disturbances.

Treat gaze stabilization as a predictive temporal inference problem rather than a post-processing filter. A seq2seq model leverages historical gaze trajectories from the saccadic search phase to predict optimized, target-centered fixation points in the subsequent fixation phase. The model captures shared temporal patterns across individuals through TimesNet-based periodic modeling and a hybrid projection mechanism.

Embedding

Embedding

Triple embedding combining token (1D conv), sinusoidal positional encoding, and timestamp linear projection to capture gaze dynamics and temporal order.

TimesNet

TimesNet

State-of-the-art temporal backbone that transforms sequences into the frequency domain to extract periodic gaze patterns via multi-scale convolutions.

Fused Project

Fused Project

Dual-branch projection combining multi-head self-attention for long-range dependencies and linear projection for local structure, fused via a learnable balance parameter.

Combined Loss

Combined Loss

MSE loss + center distance regularization + dispersion consistency, preventing global bias and over-collapse of predicted fixations.

Sliding Window

Sliding Window

Progressive auto-regressive training strategy that consumes model predictions incrementally, alleviating exposure bias in seq2seq inference.

Synthetic Data

Synthetic Data

Spatial contraction and translation toward targets to generate idealized fixation sequences, expanding training diversity and improving generalization.

Fig. 5. Architecture of TimeGazer. Raw historical gaze sequences are encoded via the embedding module (normalization, token/positional/temporal embeddings, predict linear), processed by TimesNet for temporal feature extraction, and projected through fused MHA and linear branches to generate the predicted gaze sequence.

Fig. 5. Architecture of TimeGazer. Raw historical gaze sequences are encoded via the embedding module (normalization, token/positional/temporal embeddings, predict linear), processed by TimesNet for temporal feature extraction, and projected through fused MHA and linear branches to generate the predicted gaze sequence.

Embedding Module: Gaze coordinates and velocities are standardized per channel, then projected via 1D convolution (token embedding), combined with sinusoidal positional encoding and timestamp linear embedding. A predict linear layer extends the temporal dimension from T to T+τ.

Results

User Study
0.60
ACR↑
+17.6% vs HoloLens 2 Native (0.51) Average fixation completion rate
User Study
30.58s
ATD↓
−6.1% vs HoloLens 2 Native (32.56s) Average task duration
User Study
1.87
GI↓
−12.6% vs HoloLens 2 Native (2.14) Gaze interruptions per trial
User Study
0.26
FTR↑
+13.0% vs HoloLens 2 Native (0.23) Fixation-to-threshold ratio
MRTK Selection
1.77s
ATD↓
−22.7% vs Native (2.29s), d=1.26 Target selection time
MRTK Navigation
15.74s
ATD↓
−17.4% vs Native (19.05s), d=1.99 Navigation task time
Subjective
3.93
Stability↑
+20.2% vs Native (3.27), p=0.0056 5-point Likert scale
Subjective
4.07
Sensitivity↑
+32.6% vs Native (3.07), p=0.0003 5-point Likert scale
Quantitative
6.41
CI↑
Concentration improvement over raw gaze Prediction horizon τ=64
Quantitative
1.41
AI↑
Accuracy improvement over raw gaze Prediction horizon τ=64
Quantitative
0.050
AD↓
Average deviation to target (meters) Robust across individuals
MetricTimeGazerHoloLens 2 Nativep-valueEffect Size
ACR ↑0.60 ± 0.230.51 ± 0.250.0034**0.873
ATD(s) ↓30.58 ± 5.0432.56 ± 5.140.035*0.917
GI(count/trial) ↓1.87 ± 0.482.14 ± 0.710.035*0.397
FTR ↑0.26 ± 0.100.23 ± 0.0840.035*0.416
Stability ↑ ours3.93 ± 0.423.27 ± 0.460.0056**1.234
Sensitivity ↑ ours4.07 ± 0.463.07 ± 0.460.0003***4.009

TimeGazer was deployed on Microsoft HoloLens 2 and evaluated with 27 participants in a counterbalanced user study. It significantly outperformed the native HoloLens 2 eye-tracking algorithm on all four metrics (ACR, ATD, GI, FTR) with p < 0.05. MRTK scenario evaluations further confirmed improvements in target selection (d=1.26) and navigation (d=1.99).

Qualitative Results

Fig. 7. User study and MRTK-based evaluation scenarios: (a) controlled AR fixation task, (b) target selection, and (c) navigation. TimeGazer significantly outperforms native HoloLens 2 eye-tracking across all interaction tasks.

Fig. 7. User study and MRTK-based evaluation scenarios: (a) controlled AR fixation task, (b) target selection, and (c) navigation. TimeGazer significantly outperforms native HoloLens 2 eye-tracking across all interaction tasks.

Fig. 4. A typical gaze trajectory example showing the transition from rapid saccadic search to stable fixation, illustrating the dispersion and deviation that TimeGazer aims to correct.

Fig. 4. A typical gaze trajectory example showing the transition from rapid saccadic search to stable fixation, illustrating the dispersion and deviation that TimeGazer aims to correct.

Results

Takeaway

Gaze trajectories encode rich temporal cues about user intent and attentional stability; exploiting these cues through seq2seq temporal modeling unlocks performance gains unattainable through static or geometry-only approaches.

Citation

@misc{xia2025timegazertemporalmodelingpredictive,
      title={TimeGazer: Temporal Modeling of Predictive Gaze Stabilization for AR Interaction}, 
      author={Yaozheng Xia and Zaiping Zhu and Bo Pang and Shaorong Wang and Sheng Li},
      year={2025},
      eprint={2510.01561},
      archivePrefix={arXiv},
      primaryClass={cs.HC},
      url={https://arxiv.org/abs/2510.01561}, 
}